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Image Denoising Using Low Rank Minimization 
With Modified Noise Estimation 

Zahid Hussain Shamsi, Hyun Sook Oh and Dai-Gyoung Kim* 


Abstract —Recently, the application of low rank minimization 
to image denoising has shown remarkable denoising results which 
are equivalent or better than those of the existing state-of-the-art 
algorithms. However, due to iterative nature of low rank opti¬ 
mization, estimation of residual noise is an essential requirement 
after each iteration. Currently, this noise is estimated by using 
the filtered noise in the previous iteration without considering 
the geometric structure of the given image. This estimate may 
be affected in the presence of moderate and severe levels of noise. 
To obtain a more reliable estimate of residual noise, we propose 
a modified algorithm (GWNNM) which includes the contribution 
of the geometric structure of an image to the existing noise 
estimation. Furthermore, the proposed algorithm exploits the 
difference of large and small singular values to enhance the edges 
and textures during the denoising process. Consequently, the 
proposed modifications achieve significant improvements in the 
denoising results of the existing low rank optimization algorithms. 

Index Terms —Low rank minimization, nuclear norm, singular 
value decomposition, residual noise estimation, denoising. 


I. Introduction 

P ATCH based image denoising in conjunction with the 
notion of non-locality has led to several state-of-the- 
art algorithms 0-0. These algorithms exploit certain prior 
knowledge like image redundancy and sparse representation 
of images in suitable transform domains. These assumptions 
are found to be generally true in case of natural images and 
greatly improve the denoising results. 

In addition to sparsity and redundancy priors, a low rank 
assumption has also been proposed recently in |5J, |6j. Ac¬ 
cording to low rank prior, a matrix constructed by stacking 
non-local similar patches from a given noisy image resides 
in a low dimensional subspace of a high dimensional space 
and therefore satisfies the low rank criterion. Dong et al. [|5J 
introduced an iterative soft thresholding scheme (SAIST) using 
Li .2 grouped sparsity regularization to penalize the singular 
values of the low rank matrices. More recently, Gu et al. 
|6] have further explored the low rank prior using weighted 
nuclear norm minimization (WNNM). This algorithm exploits 
the physical significance of the singular values and treats 
them differently according to their respective magnitudes. 
Both low rank minimization algorithms 0-0 have produced 
outstanding denoising results by exploiting low rank prior 
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confirming to its validity as a suitable assumption for image 
restoration problem. 

In this paper, we investigate WNNM algorithm for its 
noise estimation scheme. The motivation to consider the noise 
estimation relies on the fact that the residual noise is one of the 
key factors for subsequent computation of singular values and 
corresponding threshold weights which are the indispensable 
ingredients of low rank minimization schemes,in particular, for 
WNNM algorithm. WNNM algorithm estimates residual noise 
by comparing the grouped patch matrices in a given noisy 
image with the corresponding patch matrices in its denoised 
version obtained in the previous iteration. In practice, this 
difference is intuitively assumed to be the noise. However, 
this assumption may not be generally true, in particular, for 
the images with the complex geometric structures j7j. 

In order to obtain a more reliable estimate of residual noise, 
we note that the geometric structure of the image should also 
be taken into account. To the best of our knowledge, this 
aspect of residual noise estimation for WNNM algorithm has 
remained unnoticed up to now. By considering the proposed 
contribution to the existing residual noise estimation, we have 
obtained remarkable improvements in the denoising results of 
the original WNNM algorithm for most of the test images. 
In particular, for moderate and high levels of noise these 
improvements are quite significant. 

We also propose to reinforce edges and texture during 
iterative denoising by splitting the singular value decomposi¬ 
tion (SVD) into low and high components. Subsequently, two 
denoised images are obtained using low and high components 
of the SVD, respectively. The difference of these two images 
is used as feedback to reinforce edges and textural regions 
during iterative denoising process. This modification further 
enhances the denoising capability of the proposed algorithm. 

The rest of the paper is organized as follows. We briefly 
describe low rank minimization algorithm for image denoising 
in Sec. [II] The proposed algorithm and its implementation are 
presented in Sec . |Hl| Experimental settings and results are 
described in Sec.|IV| Finally, conclusions are drawn in Sec.[V] 

II. Low Rank Minimization for image denoising 

We consider the well known image denoising problem 

y = x + n, (1) 

where x is the original image to be recovered from the noisy 
observed image y and n ~ jV (0, <7„I) is an additive white 
Gaussian noise with known standard deviation a n . In terms 
of patch based formulation, non-local similar patches are 
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searched for local patch p yj at each pixel location j in the 
observed image. Afterwards, the patches similar to a patch 
p yj can be stacked to construct a matrix M yj . Using patch 
based formulation, the original problem (|T]i can be expressed 
as (6) 

M y . =M X .+M n ., (2) 


where M Xj and M nj denote the corresponding patch matrices 
of original image and additive noise, respectively. 

The patch matrix M Xj defined in (J2l) is intuitively assumed 
to be a low rank matrix because of the image redundancy 
and similarity prior 0. Thus, a noise free patch can be 
recovered by using low rank minimization. However, low 
rank minimization is a non-convex NP hard problem |0 
0- Therefore, nuclear norm minimization (NNM) has been 
heuristically proposed as its convex regularization which can 
be written as 


M x . = arg min -^r 
J ^M Xj 0-2 


M y .-M x . 


M x . 

X J 


(3) 


where ||.= Li A,- (•) is the nuclear norm defined using 
the singular values A,- of the matrix M Xj and ||.|| f denotes 
Frobenius norm. Convexity of the reformulated problem ([0 
assures the global optimal solution through singular value 
decomposition [0 

UAV t = svd ^M x .^ , 

Ae = Sg (A) = max(A; — 0,0), (4) 


where 9 denotes a soft threshold applied to all the singular 
values A, of the matrix M XJ without considering the importance 
of the large and small singular values. 

To consider physical significance of singular values, Gu 
et al. |6:| proposed weighted nuclear norm minimization 
(WNNM) algorithm which replaces || -||* by ||.|| wt in (|3j» where 
ll-IL* = L; |vv,A, (•)( i s the weighted nuclear norm and the 
weights Wi are defined as (6) 


Wi = 


Csjm 

(A^r 


(5) 


where c > 0 is a constant and m is the number of non¬ 
local patches similar to the given patch p yj . To avoid possible 
division by zero, we set the constant e = 10 16 . Note that in 
this case, the solution is similar to ([dji with the exception that 
A g is now written as 


A w = S w (A) = max (A,- - w t , 0). (6) 

It can be observed from 0 and 0- the larger singular 
values are less penalized as compared to the smaller ones in 
accordance with the basic requirement of the image denoising. 
However, the global optimal solution is not guaranteed because 
WNNM becomes a non-convex problem due to the non¬ 
descending threshold weights w,- 0- Nevertheless, Gu et 
al. |6| have shown that iteratively, this optimization scheme 
reaches its local minimum fixed point solution. 


III. Proposed Algorithm 
In the following, we describe how the existing residual 
noise estimation used in 0, © can be further enhanced by 
considering the proposed modification. 


A. Proposed Noise Estimation Method 


Due to the pivotal role of residual noise, it is highly 
desirable to quantify the amount of noise left in the current 
denoised image for further processing in the next iteration. 
This remaining noise is termed as residual noise in this paper. 
The variance of the residual noise for next iteration ik + \ ) 
can be defined as the difference between the variance of the 
initial given white Gaussian noise and that of the filtered 

noise at previous iteration k. In WNNM algorithm, 

the variance of the filtered noise is estimated by comparing 
grouped patches matrix at location j in the given noisy image 
y and the corresponding matrix in its previously denoised 
version yW. For simplicity, the variance of filtered noised can 
be expressed at image level as 0 



y-y w 


(7) 


By using Q, Dong et al. Q proposed an intuitive estimate 
for the standard deviation of the residual noise as 



where ales * * denotes the estimated standard deviation of the 
residual noise present at ( k+l) th iteration. The constant y> 
0 is a scaling factor, heuristically introduced to control the 
re-estimation of the standard deviation [|3J. Using this noise 
estimate, the singular values of local patch matrix M yj ^ +1 ) 
are adjusted as |0 



It is important to note that the noise estimation approach 
0 utilizes I 2 norm 0 which is sensitive to outliers, more 
specifically, in the presence of severe noise. More importantly, 
the difference ([7]) between a given noisy image and its filtered 
version is intuitively taken as noise. However, this assumption 
may not be generally true, in particular, for the images with 
the complex geometric structures [7j. In fact, the filtered 
noise also contains some geometric structure which is lost 
during denoising. Thus, the above expression of residual noise 
0 needs certain modification to account for the geometric 
details of the previously denoised image for residual noise 
estimation. In order to identify the image structure in terms of 
patches, Zhu and Milanfar [(TO) proposed the diagonalization 
of gradient covariance matrix C for each patch p yj . The 
covariance matrix C is defined as © 



8x{l)gy{l) \ 

8 2 y(l) J 


( 10 ) 


where [gx(l),gy(0} T denotes the gradient vector at location l 
within the patch p Vj . The large and small eigenvalues of the 
covariance matrix C represent the directions of maximum and 
minimum variations in the image structure, respectively. Yan- 
Li et al. |7 | utilized this concept of gradient covariance ma¬ 
trices to identify the weak textured patches for the estimation 
of the noise variance in a given noisy image. The standard 
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(k) 

deviation, Og eom , of the noise is subsequently estimated by 
applying PCA to these weak textured patches iteratively. We 
propose to consider this geometric contribution in residual 
noise estimation as 



Finally, the modified noise estimate is expressed as a convex 
combination of geometric (111 and non-geometric contribu¬ 
tions 0 

o£ +1) = aa-,£ +1) + (1 - a)o r es +1 \ (12) 


where a € (0,1) is tuned experimentally according to the 
initially given standard deviation dA?/ = <J n . For low levels of 
initial noise (a,, < 30), we choose a = 0.9 and for moderate 
and severe noise levels (cr„ > 30), we set a = 0.8. The 
relatively smaller value of a for <J„ > 30 indicates that the 
previous noise estimation 0 is significantly affected due 
to higher levels of given noise and is subjected to larger 
correction from the geometric counterpart. Fine tuning of a 
is also possible according to different noise levels. However, 
our primary objective is to highlight the importance of the 
proposed modification. Subsequently, the singular values can 
now be adjusted by plugging into ©• 


B. Edge and Texture Enhancement 

Due to thresholding 0- some of the smaller singular values 
may eventually reduce to zero or very close to zero. However, 
we propose to utilize these values in the feedback step for 
edge and texture reinforcement as follows. We identify large 
and small singular values by setting a threshold T and then 
split the truncated matrix A w as 

A W = A"** + A(r, (13) 

where A^ lgh * and A^°" * contain only large and small singular 
values of M yj , respectively. The optimal solution (|6j can then 
be re-written as 

My. = U (A^ igh) +aI / ow) ) V T = My.^l — +—M y .( /ow ).(14) 



Figure 1: Columns from left to right show the noisy (cj„ = 50) 
and denoised images using BM3D, WNNM and GWNNM for 
house and Barbara images, respectively. 


As a result of this splitting, we construct two corresponding 
images y( hl 8 h ) and y( ,ovv ) which contain high and low energy 
components, respectively. By exploiting intrinsic nature of 
these two images, the difference image y^ hl s h ) — y( low ) can 
be associated to sharpness and contrast enhancement. Also 
the noise level is gradually reduced in these two images, 
iteratively. Thus, the edges and texture may become more 
clear in the difference image at each iteration. Therefore, 
we intuitively propose to use this difference in the feedback 
step to preserve and enhance geometric details of the image 
during iterative denoising. Experimental results confirm the 
effectiveness of the proposed modification as well. 

The summary of the proposed algorithm is provided in 
Algorithm [T] 


Algorithm 1 Image Denoising Using GWNNM 


Input: Noisy image y 


(!) 


(Jt) 

■y tow 


Initialize yW = y, y)^ = y, y'^ gh = y 
for k=l:L-l do 

y (k+i) = y (k) + § (y_ y (*A + n (y£) gh 

for each patch p yj £ y(*+0 ,Jq 

Construct patch matrix M yj using (|2|! 
Estimate residual noise a ,£ +1 ' using |l2j 
Compute [U,A,V] = SVD ^M y . j 
Compute threshold weight vector w using ([5]> 
Obtain truncated matrix A w using 0 


Split A w = A [ " w + A'^ s " using threshold T and (1 13b 


.high 


Obtain the estimates M yj Zo14 ’ and M y . hlgh using (14 1 


end for 


Aggregate M y ., M y . Zow and M y . hlgh to construct images 


y (i+1) ’ y*™ 1 and yi^ 1) . respectively 


14: end for 

Output: Clean Image y' L} 


IV. Experimental Results 

We consider eleven frequently used standard images to 
compare the proposed algorithm (GWNNM) with various 
state-of-the-art algorithms: BM3D (2}, LSSC 0, SAIST |5), 
WNNM j6:|. We use PSNR values for comparison which are 
reported in |6j. 

We use the same default values of the parameters which 
were selected in WNNM, heuristically, depending upon the 
initially given noise level [|6j. Therefore we skip the details 
of those parameters due to space limitation. However, we 
have set the size of search window (A vl , = 15 x 15) instead 
of (30 x 30) for all the images of size 512 x 512. In fact, 
block matching process in WNNM is computationally very 
expensive and in case of A w = 30 x 30 for 512 x 512 images, 
the computational cost is much greater than that of 256 x 256 
image. Furthermore, it is pointed out in (ITJ that selecting 
larger size of search window has no major benefit in denoising 
results except for pseudo-periodic images like fingerprint im¬ 
age. Therefore, for fingerprint image we use the same value of 
A w = 30 x 30 as used in WNNM. The setting of new parameter 
a has already been explained in the previous section. The other 
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Table I: Denoisig results (PSNR) of various state-of-the-art methods 



a n = 10 

On = 30 

Image 

BM3D 

LSSC 

SAIST 

WNNM 

GWNNM 

BM3D 

LSSC 

SAIST 

WNNM 

GWNNM 

C. Man 

34.18 

34.24 

34.30 

34.44 

34.45 

28.64 

28.63 

28.36 

28.80 

28.80 

House 

36.71 

36.95 

36.66 

36.95 

36.98 

32.09 

32.41 

32.30 

35.52 

35.57 

Peppers 

34.68 

34.80 

34.82 

34.95 

34.97 

29.28 

29.25 

29.24 

29.49 

29.51 

Monarch 

34.12 

34.44 

34.76 

35.03 

35.03 

28.36 

28.20 

28.65 

28.92 

28.93 

J.Bean 

37.91 

38.69 

38.37 

38.93 

38.97 

31.97 

32.39 

32.14 

32.46 

32.56 

Lena 

35.93 

35.83 

35.90 

36.03 

36.04 

31.26 

31.18 

31.27 

31.43 

31.50 

Barbara 

34.98 

34.98 

35.24 

35.51 

35.49 

29.81 

29.60 

30.14 

30.31 

30.43 

F.print 

32.46 

32.57 

32.69 

32.82 

32.82 

26.83 

26.68 

26.95 

26.99 

27.10 

Boat 

33.92 

34.01 

33.91 

34.09 

34.08 

29.12 

29.06 

28.98 

29.24 

29.21 

Hill 

33.62 

33.66 

33.65 

33.79 

33.79 

29.16 

29.09 

29.06 

29.25 

29.22 

Man 

33.98 

34.10 

34.12 

34.23 

34.25 

28.86 

28.87 

28.81 

29.00 

28.99 


= 50 

<7 n = 100 

Image 

BM3D 

LSSC 

SAIST 

WNNM 

GWNNM 

BM3D 

LSSC 

SAIST 

WNNM 

GWNNM 

C. Man 

26.12 

26.35 

26.15 

26.42 

26.43 

23.07 

23.15 

23.09 

23.36 

23.39 

House 

29.69 

29.99 

30.17 

30.32 

30.42 

25.87 

25.71 

26.53 

26.68 

26.82 

Peppers 

26.68 

26.79 

26.73 

26.91 

26.93 

23.39 

23.20 

23.32 

23.46 

23.56 

Monarch 

25.82 

25.88 

26.10 

26.32 

26.31 

22.52 

22.24 

22.61 

22.95 

22.99 

J.Bean 

29.26 

29.42 

29.32 

29.62 

29.71 

25.80 

25.64 

25.82 

26.04 

26.15 

Lena 

29.05 

28.95 

29.01 

29.24 

29.34 

25.95 

25.96 

25.93 

26.20 

26.34 

Barbara 

27.23 

27.03 

27.51 

27.79 

27.93 

23.62 

23.54 

24.07 

24.37 

24.49 

F.print 

24.53 

24.26 

24.52 

24.67 

24.76 

21.61 

21.30 

21.62 

21.81 

21.87 

Boat 

26.78 

26.77 

26.63 

26.97 

26.93 

23.97 

23.87 

23.80 

24.10 

24.15 

Hill 

27.19 

27.14 

27.04 

27.34 

27.31 

24.58 

24.47 

24.29 

24.75 

24.71 

Man 

26.81 

26.72 

26.68 

26.94 

26.92 

24.22 

23.98 

24.01 

24.36 

24.36 


two parameters T) and x are experimentally selected as 0.01 
and 0.5, respectively. 

The comparison of the denoising results is described in 
Table. [I] and the best results are highlighted using bold faced 
values. Also, a limited comparison of visual quality for BM3D, 
WNNM and GWNNM algorithms is shown in Fig. |T| at noise 
level o n = 50. As shown in Table. |T]that WNNM and GWNNM 
perform always better than the rest of the algorithms. For low 
noise (o n - 10) the performance of GWNNM is in general 
equivalent to WNNM. It is reasonable because the filtered 
noise 0 contains negligible signatures of geometric details 
and needs only a small correction from geometric estimate 
CD- However, for high noise levels (o n > 30), GWNNM gen¬ 
erally performs better than WNNM. In particular, at <7„ = 100, 
the improvements in PSNR values are greater than 0.1 dB for 
the images of house, jelly beans, peppers, Lena, Barbara and 
fingerprint. Thus, the improved results, in particular, for geo¬ 
metrically rich and complex images like Barbara, fingerprint, 
house and Lena, support our assumption that the estimation of 
residual noise needs modification due to geometric structure 
of the image. 

V. Conclusion 

We have highlighted the importance of residual noise es¬ 
timation for low rank optimization to further improve the 
results. Further we have proposed a method to make the noise 
estimate more precise by using geometrical structure of a 
given noisy image. Alternate effective approaches for noise 
estimation also exist and can be utilized. However, our primary 
intent was to indicate the deficiency in the existing residual 


noise estimation scheme which only exploits filtered noise in 
the previous iteration without considering the geometric details 
present in the filtered noise. 
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